@ArtiomTr all required changes done!

looks like client build fails? something with http_api test driver:

error[E0609]: no field `proof_data` on type `GossipProofJSON`
   --> http_api/src/test_driver.rs:463:32
    |
463 |     let proof_hex = step.proof.proof_data.data.trim_start_matches("0x");
    |                                ^^^^^^^^^^ unknown field
    |

looks like client build fails? something with http_api test driver:

error[E0609]: no field `proof_data` on type `GossipProofJSON`
   --> http_api/src/test_driver.rs:463:32
    |
463 |     let proof_hex = step.proof.proof_data.data.trim_start_matches("0x");
    |                                ^^^^^^^^^^ unknown field
    |

yeah sorry about that, I forgot to check build before committing. should be fine now.

hey @ArtiomTr @SoarinSkySagar , I was just in general looking through this and is there a reason why you guys are directly adding the JSON fixtures to the repository.
I think using the leanspec fixtures release in the Makefile and then downloading it on the local for testing would be better than adding the fixtures directly to the repo and then changing them again and again based on the fixture changes.
I have written a small review for the make file that would perform this task.

is there a reason why you guys are directly adding the JSON fixtures to the repository

@Sahilgill24 I think the main concern we have with this approach is that it takes a very long time just to regenerate the test vectors (~57 minutes for all tests and total 2.5 hours with deterministic tests, on my machine). So regenerating the test vectors each time during testing consumes a lot of time when we can just download and reuse them and only regenerate them when we have a major update on the tests.

Would like to hear your suggestions/opinion on this.

I am not sure why it would be taking 57 minutes for the regeneration because the tar archive for the test vectors is just 150Mb and running them is like almost 1-2 minutes.
are you surely running them by downloading the fixtures from the releases right? rather than using the uv sync command on the leanspec repo, because I think that can take so much time

hey ＠ArtiomTr ＠SoarinSkySagar , I was just in general looking through this and is there a reason why you guys are directly adding the JSON fixtures to the repository.

Because we don't want to make our CI builds fragile. See, with your proposed script, you're downloading "latest" release. This makes CI impure, and this brings tons of issues to development process:

• if I want to debug test failure, how do I make sure my CI downloads exact same version as I do?
• if someone accidentally breaks test vectors / pushes incorrect test vectors into lean spec, how can I revert those changes to one version back?

This also makes some things more irritating, though they don't affect development directly:

• re-running CI may pass/fail on previously failing/passing commit;
• CI on old branches (like devnet-2 for example) will fail, if we try to re-run tests.

Some good things that we get from our approach, which are nice-to-have, but not the main reason we do things such way:

• git checkout <branch> automatically pulls correct version of tests - this means I can easily run cargo test -p fork_choice for example, and never see some obscure errors, something like glob pattern "test_vectors/fork_choice/*/fc/*/*.json" didn't match any files! or just randomly failing tests, because I didn't ran make test, that automatically pulls correct version of test vectors;
• the "pulling correct version of tests" thing also means that I can run tests granularly, what especially comes handy with AI agents - not needing to run whole test suite makes iterating faster;
• nothing stops us from post-processing (e.g., reshaping) or changing test vectors by hand. We haven't tried doing that, but it may be handy in some cases.

Our approach has some downsides too:

• test vectors pollute history, git database, and I think probably slow down some things, like checkouts;
• we have to not forget to sync our tests with spec version, otherwise we may fix issues that aren't present in latest spec version.

With that all said, I think our current approach is better than downloading latest spec test version. Maybe if leanSpec does tagged releases at some point, then we could transition to downloading spec tests instead of generating them on our own - though I think currently that doesn't make much sense for leanSpec, considering the pace new changes are introduced.

@Sahilgill24 if you want to continue discussion, I can create a separate issue for that, so we can discuss other approaches of managing test vectors.

@ArtiomTr surely, we can discuss a few approaches

I am not sure why it would be taking 57 minutes for the regeneration because the tar archive for the test vectors is just 150Mb and running them is like almost 1-2 minutes. are you surely running them by downloading the fixtures from the releases right? rather than using the uv sync command on the leanspec repo, because I think that can take so much time

yes, I was using the uv command